Compressed Gradient Methods With Hessian-Aided Error Compensation

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed reduce communication load at price loss solution accuracy. Recently, it shown how errors can compensated algorithm improve Even though convergence guarantees error-compensated algorithms established, there very limited theoretical support quantifying observed improvements In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation on quadratic problems. We also present strong Hessian-based compensation stochastic descent. Our numerical experiments highlight benefits and demonstrate similar are attained when only diagonal Hessian approximation used.